Problem Statement¶
Business Context¶
Workplace safety in hazardous environments like construction sites and industrial plants is crucial to prevent accidents and injuries. One of the most important safety measures is ensuring workers wear safety helmets, which protect against head injuries from falling objects and machinery. Non-compliance with helmet regulations increases the risk of serious injuries or fatalities, making effective monitoring essential, especially in large-scale operations where manual oversight is prone to errors and inefficiency.
To overcome these challenges, SafeGuard Corp plans to develop an automated image analysis system capable of detecting whether workers are wearing safety helmets. This system will improve safety enforcement, ensuring compliance and reducing the risk of head injuries. By automating helmet monitoring, SafeGuard aims to enhance efficiency, scalability, and accuracy, ultimately fostering a safer work environment while minimizing human error in safety oversight.
Objective¶
As a data scientist at SafeGuard Corp, you are tasked with developing an image classification model that classifies images into one of two categories:
- With Helmet: Workers wearing safety helmets.
- Without Helmet: Workers not wearing safety helmets.
Data Description¶
The dataset consists of 631 images, equally divided into two categories:
- With Helmet: 311 images showing workers wearing helmets.
- Without Helmet: 320 images showing workers not wearing helmets.
Dataset Characteristics:
- Variations in Conditions: Images include diverse environments such as construction sites, factories, and industrial settings, with variations in lighting, angles, and worker postures to simulate real-world conditions.
- Worker Activities: Workers are depicted in different actions such as standing, using tools, or moving, ensuring robust model learning for various scenarios.
Installing and Importing the Necessary Libraries¶
!pip install tensorflow[and-cuda] numpy==1.25.2 -q
Installing build dependencies ... done error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. Getting requirements to build wheel ... error error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip.
import tensorflow as tf
print("Num GPUs Available:", len(tf.config.list_physical_devices('GPU')))
print(tf.__version__)
Num GPUs Available: 1 2.19.0
Note:
After running the above cell, kindly restart the notebook kernel (for Jupyter Notebook) or runtime (for Google Colab) and run all cells sequentially from the next cell.
On executing the above line of code, you might see a warning regarding package dependencies. This error message can be ignored as the above code ensures that all necessary libraries and their dependencies are maintained to successfully execute the code in this notebook.
import os
import random
import numpy as np # Importing numpy for Matrix Operations
import pandas as pd
import seaborn as sns
import matplotlib.image as mpimg # Importing pandas to read CSV files
import matplotlib.pyplot as plt # Importting matplotlib for Plotting and visualizing images
import math # Importing math module to perform mathematical operations
import cv2
# Tensorflow modules
import keras
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator # Importing the ImageDataGenerator for data augmentation
from tensorflow.keras.models import Sequential # Importing the sequential module to define a sequential model
from tensorflow.keras.layers import Dense,Dropout,Flatten,Conv2D,MaxPooling2D,BatchNormalization # Defining all the layers to build our CNN Model
from tensorflow.keras.optimizers import Adam,SGD # Importing the optimizers which can be used in our model
from sklearn import preprocessing # Importing the preprocessing module to preprocess the data
from sklearn.model_selection import train_test_split # Importing train_test_split function to split the data into train and test
from sklearn.metrics import confusion_matrix
from tensorflow.keras.models import Model
from keras.applications.vgg16 import VGG16 # Importing confusion_matrix to plot the confusion matrix
# Display images using OpenCV
from google.colab.patches import cv2_imshow
#Imports functions for evaluating the performance of machine learning models
from sklearn.metrics import confusion_matrix, f1_score,accuracy_score, recall_score, precision_score, classification_report
from sklearn.metrics import mean_squared_error as mse # Importing cv2_imshow from google.patches to display images
# Ignore warnings
import warnings
warnings.filterwarnings('ignore')
# Set the seed using keras.utils.set_random_seed. This will set:
# 1) `numpy` seed
# 2) backend random seed
# 3) `python` random seed
tf.keras.utils.set_random_seed(812)
Data Overview¶
Loading the data¶
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
images = np.load('/content/drive/MyDrive/Colab Notebooks/helmet/data/images_proj.npy')
labels_df = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/helmet/data/Labels_proj.csv')
print("Dataset Information:")
print(f"Images shape: {images.shape}")
print(f"Images dtype: {images.dtype}")
print(f"Images min value: {images.min()}, max value: {images.max()}")
print(f"Number of labels: {len(labels_df)}")
print(f"Labels shape: {labels_df.shape}")
# Display first few labels
print("\nFirst 10 labels:")
print(labels_df.head(10))
# Convert labels to numpy array for easier handling
labels = labels_df['Label'].values
print(f"\nUnique labels: {np.unique(labels)}")
print(f"Label distribution:")
print(f"Class 0 (Without Helmet): {np.sum(labels == 0)} images")
print(f"Class 1 (With Helmet): {np.sum(labels == 1)} images")
Dataset Information: Images shape: (631, 200, 200, 3) Images dtype: uint8 Images min value: 0, max value: 255 Number of labels: 631 Labels shape: (631, 1) First 10 labels: Label 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 1 Unique labels: [0 1] Label distribution: Class 0 (Without Helmet): 320 images Class 1 (With Helmet): 311 images
Exploratory Data Analysis¶
Plot random images from each of the classes and print their corresponding labels.¶
def plot_sample_images(images, labels, n_samples=8):
"""
Plot random sample images from each class
"""
fig, axes = plt.subplots(2, n_samples, figsize=(16, 6))
# Get indices for each class
class_0_indices = np.where(labels == 0)[0]
class_1_indices = np.where(labels == 1)[0]
# Randomly sample indices
random_class_0 = np.random.choice(class_0_indices, n_samples, replace=False)
random_class_1 = np.random.choice(class_1_indices, n_samples, replace=False)
# Plot class 0 (Without Helmet)
for i, idx in enumerate(random_class_0):
axes[0, i].imshow(images[idx])
axes[0, i].axis('off')
if i == 0:
axes[0, i].set_title('Without Helmet', fontsize=12, fontweight='bold')
axes[0, i].set_title(f'Index: {idx}', fontsize=8)
# Plot class 1 (With Helmet)
for i, idx in enumerate(random_class_1):
axes[1, i].imshow(images[idx])
axes[1, i].axis('off')
if i == 0:
axes[1, i].set_title('With Helmet', fontsize=12, fontweight='bold')
axes[1, i].set_title(f'Index: {idx}', fontsize=8)
plt.suptitle('Sample Images from Each Class', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()
# Plot sample images
plot_sample_images(images, labels, n_samples=8)
Checking for class imbalance¶
# Visualize class distribution
plt.figure(figsize=(12, 5))
# Bar plot
plt.subplot(1, 2, 1)
class_counts = pd.Series(labels).value_counts().sort_index()
bars = plt.bar(['Without Helmet (0)', 'With Helmet (1)'], class_counts.values,
color=['#ff6b6b', '#4ecdc4'])
plt.ylabel('Number of Images')
plt.title('Class Distribution')
for bar, count in zip(bars, class_counts.values):
plt.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 5,
str(count), ha='center', va='bottom', fontweight='bold')
# Pie chart
plt.subplot(1, 2, 2)
plt.pie(class_counts.values, labels=['Without Helmet', 'With Helmet'],
autopct='%1.1f%%', colors=['#ff6b6b', '#4ecdc4'], startangle=90)
plt.title('Class Distribution (Percentage)')
plt.suptitle('Dataset Class Balance Analysis', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()
# Print imbalance ratio
imbalance_ratio = class_counts[0] / class_counts[1]
print(f"Class imbalance ratio (Without/With): {imbalance_ratio:.2f}")
print(f"The dataset is {'balanced' if 0.9 <= imbalance_ratio <= 1.1 else 'slightly imbalanced'}")
Class imbalance ratio (Without/With): 1.03 The dataset is balanced
Data Preprocessing¶
Converting images to grayscale¶
def convert_to_grayscale(images):
"""
Convert RGB images to grayscale using weighted average method
"""
# Create array to store grayscale images
grayscale_images = np.zeros((images.shape[0], images.shape[1], images.shape[2], 1),
dtype=images.dtype)
# Convert each image to grayscale
for i in range(images.shape[0]):
# Use cv2 for proper grayscale conversion
gray = cv2.cvtColor(images[i], cv2.COLOR_RGB2GRAY)
grayscale_images[i] = gray.reshape(images.shape[1], images.shape[2], 1)
return grayscale_images
# Convert images to grayscale
grayscale_images = convert_to_grayscale(images)
print(f"Original images shape: {images.shape}")
print(f"Grayscale images shape: {grayscale_images.shape}")
# Visualize conversion
fig, axes = plt.subplots(2, 4, figsize=(12, 6))
sample_indices = np.random.choice(len(images), 4, replace=False)
for i, idx in enumerate(sample_indices):
# Original images
axes[0, i].imshow(images[idx])
axes[0, i].axis('off')
axes[0, i].set_title(f'Original (Index: {idx})')
# Grayscale images
axes[1, i].imshow(grayscale_images[idx].squeeze(), cmap='gray')
axes[1, i].axis('off')
axes[1, i].set_title(f'Grayscale (Label: {labels[idx]})')
plt.suptitle('RGB to Grayscale Conversion', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()
Original images shape: (631, 200, 200, 3) Grayscale images shape: (631, 200, 200, 1)
Splitting the dataset¶
from sklearn.model_selection import train_test_split
# First split: 80% train+val, 20% test
X_temp, X_test, y_temp, y_test = train_test_split(
grayscale_images, labels,
test_size=0.2,
random_state=42,
stratify=labels
)
# Second split: From the 80%, take 75% for train (60% of total) and 25% for val (20% of total)
X_train, X_val, y_train, y_val = train_test_split(
X_temp, y_temp,
test_size=0.25,
random_state=42,
stratify=y_temp
)
# Convert labels to pandas Series for consistency
y_train = pd.Series(y_train)
y_val = pd.Series(y_val)
y_test = pd.Series(y_test)
print("Dataset split:")
print(f"Training set: {X_train.shape[0]} images ({X_train.shape[0]/len(grayscale_images)*100:.1f}%)")
print(f"Validation set: {X_val.shape[0]} images ({X_val.shape[0]/len(grayscale_images)*100:.1f}%)")
print(f"Test set: {X_test.shape[0]} images ({X_test.shape[0]/len(grayscale_images)*100:.1f}%)")
print(f"\nClass distribution in splits:")
print(f"Train - Class 0: {(y_train==0).sum()}, Class 1: {(y_train==1).sum()}")
print(f"Val - Class 0: {(y_val==0).sum()}, Class 1: {(y_val==1).sum()}")
print(f"Test - Class 0: {(y_test==0).sum()}, Class 1: {(y_test==1).sum()}")
Dataset split: Training set: 378 images (59.9%) Validation set: 126 images (20.0%) Test set: 127 images (20.1%) Class distribution in splits: Train - Class 0: 192, Class 1: 186 Val - Class 0: 64, Class 1: 62 Test - Class 0: 64, Class 1: 63
Data Normalization¶
X_train = X_train.astype('float32') / 255.0
X_val = X_val.astype('float32') / 255.0
X_test = X_test.astype('float32') / 255.0
print("Data normalization completed:")
print(f"X_train min/max: {X_train.min():.3f}/{X_train.max():.3f}")
print(f"X_val min/max: {X_val.min():.3f}/{X_val.max():.3f}")
print(f"X_test min/max: {X_test.min():.3f}/{X_test.max():.3f}")
# Verify shapes
print(f"\nFinal shapes:")
print(f"X_train: {X_train.shape}, y_train: {y_train.shape}")
print(f"X_val: {X_val.shape}, y_val: {y_val.shape}")
print(f"X_test: {X_test.shape}, y_test: {y_test.shape}")
Data normalization completed: X_train min/max: 0.000/1.000 X_val min/max: 0.000/1.000 X_test min/max: 0.000/1.000 Final shapes: X_train: (378, 200, 200, 1), y_train: (378,) X_val: (126, 200, 200, 1), y_val: (126,) X_test: (127, 200, 200, 1), y_test: (127,)
Model Building¶
Model Evaluation Criterion¶
Utility Functions¶
# defining a function to compute different metrics to check performance of a classification model built using statsmodels
# def model_performance_classification(model, predictors, target):
# """
# Function to compute different metrics to check classification model performance
# model: classifier
# predictors: independent variables (image data)
# target: dependent variable
# """
# # checking which probabilities are greater than threshold
# pred = model.predict(predictors).reshape(-1) > 0.5
# target = target.to_numpy().reshape(-1)
# acc = accuracy_score(target, pred) # to compute Accuracy
# recall = recall_score(target, pred, average='weighted') # to compute Recall
# precision = precision_score(target, pred, average='weighted') # to compute Precision
# f1 = f1_score(target, pred, average='weighted') # to compute F1-score
# # creating a dataframe of metrics
# df_perf = pd.DataFrame({"Accuracy": acc, "Recall": recall, "Precision": precision, "F1 Score": f1,},index=[0],)
# return df_perf
def model_performance_classification(model, predictors, target):
"""
Function to compute different metrics to check classification model performance
model: classifier
predictors: independent variables
target: dependent variable
"""
# checking which probabilities are greater than threshold
pred = model.predict(predictors).reshape(-1)>0.5
target = target.to_numpy().reshape(-1)
acc = accuracy_score(target, pred) # to compute Accuracy
recall = recall_score(target, pred, average='weighted') # to compute Recall
precision = precision_score(target, pred, average='weighted') # to compute Precision
f1 = f1_score(target, pred, average='weighted') # to compute F1-score
# creating a dataframe of metrics
df_perf = pd.DataFrame({"Accuracy": acc, "Recall": recall, "Precision": precision, "F1 Score": f1,},index=[0],)
return df_perf
def plot_confusion_matrix(model,predictors,target,ml=False):
"""
Function to plot the confusion matrix
model: classifier
predictors: independent variables
target: dependent variable
ml: To specify if the model used is an sklearn ML model or not (True means ML model)
"""
# checking which probabilities are greater than threshold
pred = model.predict(predictors).reshape(-1)>0.5
target = target.to_numpy().reshape(-1)
# Plotting the Confusion Matrix using confusion matrix() function which is also predefined tensorflow module
confusion_matrix = tf.math.confusion_matrix(target,pred)
f, ax = plt.subplots(figsize=(10, 8))
sns.heatmap(
confusion_matrix,
annot=True,
linewidths=.4,
fmt="d",
square=True,
ax=ax
)
plt.show()
Model 1: Simple Convolutional Neural Network (CNN)¶
def build_simple_cnn(input_shape):
"""
Build a simple CNN architecture for helmet detection
"""
model = Sequential([
# First Convolutional Block
Conv2D(32, (3, 3), activation='relu', padding='same', input_shape=input_shape),
BatchNormalization(),
Conv2D(32, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.25),
# Second Convolutional Block
Conv2D(64, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.25),
# Third Convolutional Block
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.4),
# Fully Connected Layers
Flatten(),
Dense(256, activation='relu'),
BatchNormalization(),
Dropout(0.5),
Dense(128, activation='relu'),
BatchNormalization(),
Dropout(0.5),
Dense(1, activation='sigmoid') # Binary classification
])
return model
# Build and compile the model
model1 = build_simple_cnn(input_shape=(200, 200, 1))
# Compile with appropriate settings
model1.compile(
optimizer=Adam(learning_rate=0.001),
loss='binary_crossentropy',
metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)
# Display model architecture
print("Model 1: Simple CNN Architecture")
model1.summary()
# Train the model
history1 = model1.fit(
X_train, y_train,
validation_data=(X_val, y_val),
epochs=30,
batch_size=32,
verbose=1,
callbacks=[
tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),
tf.keras.callbacks.ReduceLROnPlateau(patience=3, factor=0.5, min_lr=1e-6)
]
)
# Evaluate Model 1
print("\n" + "="*50)
print("Model 1 Performance Evaluation")
print("="*50)
train_perf1 = model_performance_classification(model1, X_train, y_train)
val_perf1 = model_performance_classification(model1, X_val, y_val)
print("Training Performance:")
print(train_perf1)
print("\nValidation Performance:")
print(val_perf1)
Model 1: Simple CNN Architecture
Model: "sequential_2"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ conv2d_12 (Conv2D) │ (None, 200, 200, 32) │ 320 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_16 │ (None, 200, 200, 32) │ 128 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_13 (Conv2D) │ (None, 200, 200, 32) │ 9,248 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_17 │ (None, 200, 200, 32) │ 128 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d_6 (MaxPooling2D) │ (None, 100, 100, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_10 (Dropout) │ (None, 100, 100, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_14 (Conv2D) │ (None, 100, 100, 64) │ 18,496 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_18 │ (None, 100, 100, 64) │ 256 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_15 (Conv2D) │ (None, 100, 100, 64) │ 36,928 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_19 │ (None, 100, 100, 64) │ 256 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d_7 (MaxPooling2D) │ (None, 50, 50, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_11 (Dropout) │ (None, 50, 50, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_16 (Conv2D) │ (None, 50, 50, 128) │ 73,856 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_20 │ (None, 50, 50, 128) │ 512 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_17 (Conv2D) │ (None, 50, 50, 128) │ 147,584 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_21 │ (None, 50, 50, 128) │ 512 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d_8 (MaxPooling2D) │ (None, 25, 25, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_12 (Dropout) │ (None, 25, 25, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten_2 (Flatten) │ (None, 80000) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_6 (Dense) │ (None, 256) │ 20,480,256 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_22 │ (None, 256) │ 1,024 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_13 (Dropout) │ (None, 256) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_7 (Dense) │ (None, 128) │ 32,896 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_23 │ (None, 128) │ 512 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_14 (Dropout) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_8 (Dense) │ (None, 1) │ 129 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 20,803,041 (79.36 MB)
Trainable params: 20,801,377 (79.35 MB)
Non-trainable params: 1,664 (6.50 KB)
Epoch 1/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 27s 1s/step - accuracy: 0.8050 - loss: 0.4329 - precision_2: 0.7927 - recall_2: 0.8480 - val_accuracy: 0.5079 - val_loss: 2.9206 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 0.0010 Epoch 2/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 21s 128ms/step - accuracy: 0.9753 - loss: 0.1011 - precision_2: 0.9948 - recall_2: 0.9575 - val_accuracy: 0.5079 - val_loss: 6.0638 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 0.0010 Epoch 3/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 126ms/step - accuracy: 0.9855 - loss: 0.0381 - precision_2: 0.9987 - recall_2: 0.9734 - val_accuracy: 0.5079 - val_loss: 5.9870 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 0.0010 Epoch 4/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 126ms/step - accuracy: 0.9908 - loss: 0.0428 - precision_2: 0.9941 - recall_2: 0.9883 - val_accuracy: 0.5079 - val_loss: 3.6192 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 0.0010 Epoch 5/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 131ms/step - accuracy: 0.9919 - loss: 0.0238 - precision_2: 0.9949 - recall_2: 0.9896 - val_accuracy: 0.5079 - val_loss: 4.3336 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 5.0000e-04 Epoch 6/30 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 126ms/step - accuracy: 1.0000 - loss: 0.0131 - precision_2: 1.0000 - recall_2: 1.0000 - val_accuracy: 0.5079 - val_loss: 4.7845 - val_precision_2: 0.0000e+00 - val_recall_2: 0.0000e+00 - learning_rate: 5.0000e-04 ================================================== Model 1 Performance Evaluation ================================================== 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 250ms/step 4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 189ms/step Training Performance: Accuracy Recall Precision F1 Score 0 0.507937 0.507937 0.257999 0.342189 Validation Performance: Accuracy Recall Precision F1 Score 0 0.507937 0.507937 0.257999 0.342189
Vizualizing the predictions¶
def visualize_predictions(model, X_data, y_true, model_name="Model", n_samples=8):
"""
Visualize model predictions on sample images
"""
# Get predictions
predictions = model.predict(X_data[:n_samples])
pred_classes = (predictions > 0.5).astype(int).reshape(-1)
# Plot
fig, axes = plt.subplots(2, 4, figsize=(12, 6))
axes = axes.ravel()
for i in range(min(n_samples, 8)):
axes[i].imshow(X_data[i].squeeze(), cmap='gray')
axes[i].axis('off')
true_label = "With Helmet" if y_true.iloc[i] == 1 else "Without Helmet"
pred_label = "With Helmet" if pred_classes[i] == 1 else "Without Helmet"
confidence = predictions[i][0] if pred_classes[i] == 1 else 1 - predictions[i][0]
color = 'green' if y_true.iloc[i] == pred_classes[i] else 'red'
axes[i].set_title(f'True: {true_label}\nPred: {pred_label} ({confidence:.2%})',
color=color, fontsize=9)
plt.suptitle(f'{model_name} - Sample Predictions', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()
# Visualize predictions for Model 1
visualize_predictions(model1, X_val, y_val, "Simple CNN")
plot_confusion_matrix(model1, X_val, y_val)
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step
Model 2: (VGG-16 (Base))¶
X_train_rgb = np.repeat(X_train, 3, axis=-1)
X_val_rgb = np.repeat(X_val, 3, axis=-1)
X_test_rgb = np.repeat(X_test, 3, axis=-1)
# Build VGG16 base model
def build_vgg16_base(input_shape):
"""
Build VGG16 model with frozen base layers
"""
# Load pre-trained VGG16
base_model = VGG16(
weights='imagenet',
include_top=False,
input_shape=input_shape,
pooling='avg'
)
# Freeze base model layers
base_model.trainable = False
# Build model
model = Sequential([
base_model,
Dense(1, activation='sigmoid')
])
return model
# Build and compile Model 2
model2 = build_vgg16_base(input_shape=(200, 200, 3))
model2.compile(
optimizer=Adam(learning_rate=0.001),
loss='binary_crossentropy',
metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)
print("Model 2: VGG16 Base Architecture")
model2.summary()
# Train Model 2
history2 = model2.fit(
X_train_rgb, y_train,
validation_data=(X_val_rgb, y_val),
epochs=20,
batch_size=32,
verbose=1,
callbacks=[
tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),
tf.keras.callbacks.ReduceLROnPlateau(patience=3, factor=0.5, min_lr=1e-6)
]
)
# Evaluate Model 2
print("\n" + "="*50)
print("Model 2 Performance Evaluation")
print("="*50)
train_perf2 = model_performance_classification(model2, X_train_rgb, y_train)
val_perf2 = model_performance_classification(model2, X_val_rgb, y_val)
print("Training Performance:")
print(train_perf2)
print("\nValidation Performance:")
print(val_perf2)
Model 2: VGG16 Base Architecture
Model: "sequential_7"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ vgg16 (Functional) │ (None, 512) │ 14,714,688 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_13 (Dense) │ (None, 1) │ 513 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 14,715,201 (56.13 MB)
Trainable params: 513 (2.00 KB)
Non-trainable params: 14,714,688 (56.13 MB)
Epoch 1/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 9s 571ms/step - accuracy: 0.5335 - loss: 0.6916 - precision_7: 0.5398 - recall_7: 0.4277 - val_accuracy: 0.7460 - val_loss: 0.6298 - val_precision_7: 0.6630 - val_recall_7: 0.9839 - learning_rate: 0.0010 Epoch 2/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 205ms/step - accuracy: 0.8451 - loss: 0.6099 - precision_7: 0.7962 - recall_7: 0.9384 - val_accuracy: 0.9762 - val_loss: 0.5558 - val_precision_7: 0.9538 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 3/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 202ms/step - accuracy: 0.9810 - loss: 0.5427 - precision_7: 0.9846 - recall_7: 0.9781 - val_accuracy: 1.0000 - val_loss: 0.4912 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 4/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 202ms/step - accuracy: 0.9967 - loss: 0.4840 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.4360 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 5/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 210ms/step - accuracy: 0.9967 - loss: 0.4327 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.3889 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 6/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 205ms/step - accuracy: 0.9967 - loss: 0.3885 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.3486 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 7/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 203ms/step - accuracy: 0.9967 - loss: 0.3505 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.3139 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 8/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 204ms/step - accuracy: 0.9967 - loss: 0.3179 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.2842 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 9/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 205ms/step - accuracy: 0.9967 - loss: 0.2896 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.2586 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 10/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 212ms/step - accuracy: 0.9967 - loss: 0.2651 - precision_7: 1.0000 - recall_7: 0.9938 - val_accuracy: 1.0000 - val_loss: 0.2364 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 11/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 210ms/step - accuracy: 1.0000 - loss: 0.2436 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.2170 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 12/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 207ms/step - accuracy: 1.0000 - loss: 0.2248 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.2001 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 13/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 208ms/step - accuracy: 1.0000 - loss: 0.2083 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1852 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 14/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 207ms/step - accuracy: 1.0000 - loss: 0.1936 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1720 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 15/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 296ms/step - accuracy: 1.0000 - loss: 0.1806 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1602 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 16/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 225ms/step - accuracy: 1.0000 - loss: 0.1689 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1498 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 17/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 238ms/step - accuracy: 1.0000 - loss: 0.1584 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1404 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 18/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 248ms/step - accuracy: 1.0000 - loss: 0.1490 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1319 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 19/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 226ms/step - accuracy: 1.0000 - loss: 0.1404 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1243 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 Epoch 20/20 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 209ms/step - accuracy: 1.0000 - loss: 0.1327 - precision_7: 1.0000 - recall_7: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1174 - val_precision_7: 1.0000 - val_recall_7: 1.0000 - learning_rate: 0.0010 ================================================== Model 2 Performance Evaluation ================================================== 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 229ms/step 4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 281ms/step Training Performance: Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0 Validation Performance: Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0
Visualizing the prediction:¶
visualize_predictions(model2, X_val_rgb, y_val, "VGG16 Base")
plot_confusion_matrix(model2, X_val_rgb, y_val)
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 194ms/step
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 159ms/step
Model 3: (VGG-16 (Base + FFNN))¶
def build_vgg16_ffnn(input_shape):
"""
Build VGG16 model with additional fully connected layers
"""
# Load pre-trained VGG16
base_model = VGG16(
weights='imagenet',
include_top=False,
input_shape=input_shape,
pooling='avg'
)
# Freeze base model layers initially
base_model.trainable = False
# Build model with additional layers
model = Sequential([
base_model,
Dense(512, activation='relu'),
BatchNormalization(),
Dropout(0.5),
Dense(256, activation='relu'),
BatchNormalization(),
Dropout(0.5),
Dense(128, activation='relu'),
BatchNormalization(),
Dropout(0.3),
Dense(1, activation='sigmoid')
])
return model
# Build and compile Model 3
model3 = build_vgg16_ffnn(input_shape=(200, 200, 3))
model3.compile(
optimizer=Adam(learning_rate=0.001),
loss='binary_crossentropy',
metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)
print("Model 3: VGG16 with FFNN Architecture")
model3.summary()
# Train Model 3 - Phase 1: With frozen base
history3_phase1 = model3.fit(
X_train_rgb, y_train,
validation_data=(X_val_rgb, y_val),
epochs=15,
batch_size=32,
verbose=1,
callbacks=[
tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),
tf.keras.callbacks.ReduceLROnPlateau(patience=3, factor=0.5, min_lr=1e-6)
]
)
# Fine-tune: Unfreeze last few layers of base model
model3.layers[0].trainable = True
# Freeze all layers except last 4
for layer in model3.layers[0].layers[:-4]:
layer.trainable = False
# Recompile with lower learning rate
model3.compile(
optimizer=Adam(learning_rate=0.0001),
loss='binary_crossentropy',
metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)
# Train Model 3 - Phase 2: Fine-tuning
history3_phase2 = model3.fit(
X_train_rgb, y_train,
validation_data=(X_val_rgb, y_val),
epochs=10,
batch_size=32,
verbose=1,
callbacks=[
tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),
tf.keras.callbacks.ReduceLROnPlateau(patience=3, factor=0.5, min_lr=1e-7)
]
)
# Evaluate Model 3
print("\n" + "="*50)
print("Model 3 Performance Evaluation")
print("="*50)
train_perf3 = model_performance_classification(model3, X_train_rgb, y_train)
val_perf3 = model_performance_classification(model3, X_val_rgb, y_val)
print("Training Performance:")
print(train_perf3)
print("\nValidation Performance:")
print(val_perf3)
Model 3: VGG16 with FFNN Architecture
Model: "sequential_9"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ vgg16 (Functional) │ (None, 512) │ 14,714,688 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_18 (Dense) │ (None, 512) │ 262,656 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_27 │ (None, 512) │ 2,048 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_18 (Dropout) │ (None, 512) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_19 (Dense) │ (None, 256) │ 131,328 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_28 │ (None, 256) │ 1,024 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_19 (Dropout) │ (None, 256) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_20 (Dense) │ (None, 128) │ 32,896 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_29 │ (None, 128) │ 512 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_20 (Dropout) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_21 (Dense) │ (None, 1) │ 129 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 15,145,281 (57.77 MB)
Trainable params: 428,801 (1.64 MB)
Non-trainable params: 14,716,480 (56.14 MB)
Epoch 1/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 74s 768ms/step - accuracy: 0.7325 - loss: 0.5423 - precision_10: 0.7392 - recall_10: 0.7314 - val_accuracy: 0.9921 - val_loss: 0.3455 - val_precision_10: 1.0000 - val_recall_10: 0.9839 - learning_rate: 0.0010 Epoch 2/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 205ms/step - accuracy: 0.9933 - loss: 0.0299 - precision_10: 0.9953 - recall_10: 0.9917 - val_accuracy: 1.0000 - val_loss: 0.2247 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 3/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 202ms/step - accuracy: 0.9971 - loss: 0.0137 - precision_10: 0.9943 - recall_10: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1748 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 4/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 211ms/step - accuracy: 0.9946 - loss: 0.0164 - precision_10: 1.0000 - recall_10: 0.9896 - val_accuracy: 1.0000 - val_loss: 0.1529 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 5/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 201ms/step - accuracy: 1.0000 - loss: 0.0072 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1446 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 6/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 202ms/step - accuracy: 1.0000 - loss: 0.0075 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.1386 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 7/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 202ms/step - accuracy: 1.0000 - loss: 0.0053 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.1189 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 8/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 219ms/step - accuracy: 0.9994 - loss: 0.0045 - precision_10: 0.9987 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0997 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 9/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 204ms/step - accuracy: 1.0000 - loss: 0.0037 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0891 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 10/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 204ms/step - accuracy: 1.0000 - loss: 0.0041 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0890 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 11/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 204ms/step - accuracy: 1.0000 - loss: 0.0033 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0707 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 12/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 204ms/step - accuracy: 1.0000 - loss: 0.0016 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0490 - val_precision_10: 0.9841 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 13/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 215ms/step - accuracy: 1.0000 - loss: 0.0024 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0352 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 14/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 205ms/step - accuracy: 1.0000 - loss: 0.0026 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0261 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 15/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 2s 205ms/step - accuracy: 1.0000 - loss: 0.0018 - precision_10: 1.0000 - recall_10: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0210 - val_precision_10: 1.0000 - val_recall_10: 1.0000 - learning_rate: 0.0010 Epoch 1/10 12/12 ━━━━━━━━━━━━━━━━━━━━ 16s 743ms/step - accuracy: 0.9989 - loss: 0.0068 - precision_11: 0.9978 - recall_11: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0403 - val_precision_11: 0.9841 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04 Epoch 2/10 12/12 ━━━━━━━━━━━━━━━━━━━━ 11s 222ms/step - accuracy: 1.0000 - loss: 0.0017 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0316 - val_precision_11: 0.9841 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04 Epoch 3/10 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 217ms/step - accuracy: 1.0000 - loss: 0.0018 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0143 - val_precision_11: 0.9841 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04 Epoch 4/10 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 216ms/step - accuracy: 1.0000 - loss: 0.0014 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0082 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04 Epoch 5/10 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 223ms/step - accuracy: 1.0000 - loss: 0.0018 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0102 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04 Epoch 6/10 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 222ms/step - accuracy: 1.0000 - loss: 0.0016 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0026 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04 Epoch 7/10 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 214ms/step - accuracy: 1.0000 - loss: 9.9746e-04 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0046 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04 Epoch 8/10 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 219ms/step - accuracy: 1.0000 - loss: 8.2732e-04 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0041 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04 Epoch 9/10 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 218ms/step - accuracy: 1.0000 - loss: 6.6513e-04 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0011 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04 Epoch 10/10 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 226ms/step - accuracy: 1.0000 - loss: 6.5313e-04 - precision_11: 1.0000 - recall_11: 1.0000 - val_accuracy: 1.0000 - val_loss: 6.4915e-04 - val_precision_11: 1.0000 - val_recall_11: 1.0000 - learning_rate: 1.0000e-04 ================================================== Model 3 Performance Evaluation ================================================== 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 209ms/step 4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 302ms/step Training Performance: Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0 Validation Performance: Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0
Visualizing the predictions¶
visualize_predictions(model3, X_val_rgb, y_val, "VGG16 with FFNN")
plot_confusion_matrix(model3, X_val_rgb, y_val)
1/1 ━━━━━━━━━━━━━━━━━━━━ 2s 2s/step
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 144ms/step
Model 4: (VGG-16 (Base + FFNN + Data Augmentation)¶
In most of the real-world case studies, it is challenging to acquire a large number of images and then train CNNs.
To overcome this problem, one approach we might consider is Data Augmentation.
CNNs have the property of translational invariance, which means they can recognise an object even if its appearance shifts translationally in some way. - Taking this attribute into account, we can augment the images using the techniques listed below
- Horizontal Flip (should be set to True/False)
- Vertical Flip (should be set to True/False)
- Height Shift (should be between 0 and 1)
- Width Shift (should be between 0 and 1)
- Rotation (should be between 0 and 180)
- Shear (should be between 0 and 1)
- Zoom (should be between 0 and 1) etc.
Remember, data augmentation should not be used in the validation/test data set.
# Create data augmentation generator
train_datagen = ImageDataGenerator(
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
horizontal_flip=True,
shear_range=0.15,
zoom_range=0.15,
fill_mode='nearest'
)
# Validation data should not be augmented
val_datagen = ImageDataGenerator()
# Prepare data for generators
train_datagen.fit(X_train_rgb)
val_datagen.fit(X_val_rgb)
# Build Model 4 (same architecture as Model 3)
model4 = build_vgg16_ffnn(input_shape=(200, 200, 3))
model4.compile(
optimizer=Adam(learning_rate=0.001),
loss='binary_crossentropy',
metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)
print("Model 4: VGG16 with FFNN + Data Augmentation")
model4.summary()
# Train Model 4 with augmented data
history4 = model4.fit(
train_datagen.flow(X_train_rgb, y_train, batch_size=32),
validation_data=val_datagen.flow(X_val_rgb, y_val, batch_size=32),
epochs=25,
verbose=1,
callbacks=[
tf.keras.callbacks.EarlyStopping(patience=7, restore_best_weights=True),
tf.keras.callbacks.ReduceLROnPlateau(patience=4, factor=0.5, min_lr=1e-6)
]
)
# Fine-tune Model 4
model4.layers[0].trainable = True
for layer in model4.layers[0].layers[:-4]:
layer.trainable = False
model4.compile(
optimizer=Adam(learning_rate=0.00005),
loss='binary_crossentropy',
metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]
)
# Continue training with fine-tuning
history4_finetune = model4.fit(
train_datagen.flow(X_train_rgb, y_train, batch_size=32),
validation_data=val_datagen.flow(X_val_rgb, y_val, batch_size=32),
epochs=15,
verbose=1,
callbacks=[
tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),
tf.keras.callbacks.ReduceLROnPlateau(patience=3, factor=0.5, min_lr=1e-7)
]
)
# Evaluate Model 4
print("\n" + "="*50)
print("Model 4 Performance Evaluation")
print("="*50)
train_perf4 = model_performance_classification(model4, X_train_rgb, y_train)
val_perf4 = model_performance_classification(model4, X_val_rgb, y_val)
print("Training Performance:")
print(train_perf4)
print("\nValidation Performance:")
print(val_perf4)
Model 4: VGG16 with FFNN + Data Augmentation
Model: "sequential_10"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ vgg16 (Functional) │ (None, 512) │ 14,714,688 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_22 (Dense) │ (None, 512) │ 262,656 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_30 │ (None, 512) │ 2,048 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_21 (Dropout) │ (None, 512) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_23 (Dense) │ (None, 256) │ 131,328 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_31 │ (None, 256) │ 1,024 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_22 (Dropout) │ (None, 256) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_24 (Dense) │ (None, 128) │ 32,896 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_32 │ (None, 128) │ 512 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_23 (Dropout) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_25 (Dense) │ (None, 1) │ 129 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 15,145,281 (57.77 MB)
Trainable params: 428,801 (1.64 MB)
Non-trainable params: 14,716,480 (56.14 MB)
Epoch 1/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 17s 858ms/step - accuracy: 0.6895 - loss: 0.5919 - precision_12: 0.7045 - recall_12: 0.6987 - val_accuracy: 0.9921 - val_loss: 0.3375 - val_precision_12: 1.0000 - val_recall_12: 0.9839 - learning_rate: 0.0010 Epoch 2/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 360ms/step - accuracy: 0.9923 - loss: 0.0377 - precision_12: 0.9847 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1960 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 3/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 359ms/step - accuracy: 0.9843 - loss: 0.0422 - precision_12: 0.9905 - recall_12: 0.9767 - val_accuracy: 1.0000 - val_loss: 0.1355 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 4/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 445ms/step - accuracy: 0.9984 - loss: 0.0187 - precision_12: 0.9969 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.1086 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 5/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 365ms/step - accuracy: 1.0000 - loss: 0.0103 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0946 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 6/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 361ms/step - accuracy: 1.0000 - loss: 0.0032 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0851 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 7/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 372ms/step - accuracy: 1.0000 - loss: 0.0054 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0751 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 8/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 362ms/step - accuracy: 0.9834 - loss: 0.0525 - precision_12: 1.0000 - recall_12: 0.9668 - val_accuracy: 0.9921 - val_loss: 0.0855 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 9/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 423ms/step - accuracy: 1.0000 - loss: 0.0058 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0755 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 10/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 360ms/step - accuracy: 0.9994 - loss: 0.0046 - precision_12: 0.9987 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0628 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 11/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 357ms/step - accuracy: 1.0000 - loss: 0.0080 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0628 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 12/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 453ms/step - accuracy: 1.0000 - loss: 0.0028 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0525 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 13/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 364ms/step - accuracy: 1.0000 - loss: 0.0028 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0387 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 14/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 412ms/step - accuracy: 1.0000 - loss: 0.0080 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0454 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 15/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 460ms/step - accuracy: 1.0000 - loss: 0.0088 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0385 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 16/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 354ms/step - accuracy: 0.9783 - loss: 0.0555 - precision_12: 0.9991 - recall_12: 0.9600 - val_accuracy: 0.9921 - val_loss: 0.0464 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 17/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 355ms/step - accuracy: 1.0000 - loss: 0.0045 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0867 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 18/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 384ms/step - accuracy: 0.9947 - loss: 0.0112 - precision_12: 1.0000 - recall_12: 0.9895 - val_accuracy: 0.9921 - val_loss: 0.0473 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 19/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 361ms/step - accuracy: 0.9888 - loss: 0.0226 - precision_12: 0.9769 - recall_12: 0.9987 - val_accuracy: 0.9921 - val_loss: 0.0382 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 20/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 420ms/step - accuracy: 1.0000 - loss: 0.0076 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0167 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 21/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 356ms/step - accuracy: 1.0000 - loss: 0.0033 - precision_12: 1.0000 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0055 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 22/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 364ms/step - accuracy: 0.9843 - loss: 0.0295 - precision_12: 0.9689 - recall_12: 0.9942 - val_accuracy: 1.0000 - val_loss: 0.0031 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 23/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 457ms/step - accuracy: 0.9980 - loss: 0.0040 - precision_12: 0.9982 - recall_12: 0.9977 - val_accuracy: 1.0000 - val_loss: 0.0021 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 24/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 360ms/step - accuracy: 0.9989 - loss: 0.0026 - precision_12: 0.9977 - recall_12: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0039 - val_precision_12: 1.0000 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 25/25 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 356ms/step - accuracy: 0.9973 - loss: 0.0044 - precision_12: 0.9944 - recall_12: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0112 - val_precision_12: 0.9841 - val_recall_12: 1.0000 - learning_rate: 0.0010 Epoch 1/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 18s 889ms/step - accuracy: 0.9982 - loss: 0.0071 - precision_13: 0.9964 - recall_13: 1.0000 - val_accuracy: 0.9841 - val_loss: 0.0620 - val_precision_13: 0.9688 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05 Epoch 2/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 369ms/step - accuracy: 0.9960 - loss: 0.0078 - precision_13: 0.9987 - recall_13: 0.9934 - val_accuracy: 0.9921 - val_loss: 0.0373 - val_precision_13: 0.9841 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05 Epoch 3/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 372ms/step - accuracy: 1.0000 - loss: 0.0011 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0051 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05 Epoch 4/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 426ms/step - accuracy: 0.9777 - loss: 0.0256 - precision_13: 0.9619 - recall_13: 0.9875 - val_accuracy: 1.0000 - val_loss: 0.0057 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05 Epoch 5/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 361ms/step - accuracy: 1.0000 - loss: 0.0022 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0145 - val_precision_13: 0.9841 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05 Epoch 6/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 389ms/step - accuracy: 0.9973 - loss: 0.0097 - precision_13: 1.0000 - recall_13: 0.9944 - val_accuracy: 0.9921 - val_loss: 0.0173 - val_precision_13: 0.9841 - val_recall_13: 1.0000 - learning_rate: 5.0000e-05 Epoch 7/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 368ms/step - accuracy: 1.0000 - loss: 0.0044 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 0.9921 - val_loss: 0.0080 - val_precision_13: 0.9841 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05 Epoch 8/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 368ms/step - accuracy: 1.0000 - loss: 0.0020 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0050 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05 Epoch 9/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 458ms/step - accuracy: 1.0000 - loss: 0.0011 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0043 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05 Epoch 10/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 363ms/step - accuracy: 1.0000 - loss: 0.0050 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0039 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05 Epoch 11/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 368ms/step - accuracy: 1.0000 - loss: 0.0027 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0016 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05 Epoch 12/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 6s 467ms/step - accuracy: 1.0000 - loss: 0.0047 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 3.1216e-04 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05 Epoch 13/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 374ms/step - accuracy: 1.0000 - loss: 0.0040 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 2.7637e-04 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05 Epoch 14/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 4s 364ms/step - accuracy: 1.0000 - loss: 0.0016 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 3.0038e-04 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05 Epoch 15/15 12/12 ━━━━━━━━━━━━━━━━━━━━ 5s 433ms/step - accuracy: 1.0000 - loss: 6.8510e-04 - precision_13: 1.0000 - recall_13: 1.0000 - val_accuracy: 1.0000 - val_loss: 3.0791e-04 - val_precision_13: 1.0000 - val_recall_13: 1.0000 - learning_rate: 2.5000e-05 ================================================== Model 4 Performance Evaluation ================================================== 12/12 ━━━━━━━━━━━━━━━━━━━━ 3s 217ms/step 4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 305ms/step Training Performance: Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0 Validation Performance: Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0
Visualizing the predictions¶
visualize_predictions(model4, X_val_rgb, y_val, "VGG16 with Data Augmentation")
plot_confusion_matrix(model4, X_val_rgb, y_val)
1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 948ms/step
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 149ms/step
Model Performance Comparison and Final Model Selection¶
performance_comparison = pd.DataFrame({
'Model': ['Simple CNN', 'VGG16 Base', 'VGG16 + FFNN', 'VGG16 + FFNN + Aug'],
'Val_Accuracy': [
val_perf1['Accuracy'].values[0],
val_perf2['Accuracy'].values[0],
val_perf3['Accuracy'].values[0],
val_perf4['Accuracy'].values[0]
],
'Val_Precision': [
val_perf1['Precision'].values[0],
val_perf2['Precision'].values[0],
val_perf3['Precision'].values[0],
val_perf4['Precision'].values[0]
],
'Val_Recall': [
val_perf1['Recall'].values[0],
val_perf2['Recall'].values[0],
val_perf3['Recall'].values[0],
val_perf4['Recall'].values[0]
],
'Val_F1': [
val_perf1['F1 Score'].values[0],
val_perf2['F1 Score'].values[0],
val_perf3['F1 Score'].values[0],
val_perf4['F1 Score'].values[0]
]
})
print("="*60)
print("MODEL PERFORMANCE COMPARISON (Validation Set)")
print("="*60)
print(performance_comparison.round(4))
# Visualize comparison
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
metrics = ['Val_Accuracy', 'Val_Precision', 'Val_Recall', 'Val_F1']
colors = ['#3498db', '#e74c3c', '#2ecc71', '#f39c12']
for idx, (ax, metric) in enumerate(zip(axes.ravel(), metrics)):
bars = ax.bar(range(len(performance_comparison)),
performance_comparison[metric],
color=colors)
ax.set_xticks(range(len(performance_comparison)))
ax.set_xticklabels(performance_comparison['Model'], rotation=45, ha='right')
ax.set_ylabel('Score')
ax.set_title(metric.replace('Val_', '').replace('_', ' '))
ax.set_ylim([0, 1])
ax.grid(axis='y', alpha=0.3)
# Add value labels on bars
for bar, value in zip(bars, performance_comparison[metric]):
ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 0.01,
f'{value:.3f}', ha='center', va='bottom', fontweight='bold')
plt.suptitle('Model Performance Comparison', fontsize=16, fontweight='bold')
plt.tight_layout()
plt.show()
# Select best model based on F1 score
best_model_idx = performance_comparison['Val_F1'].idxmax()
best_model_name = performance_comparison.loc[best_model_idx, 'Model']
best_model = [model1, model2, model3, model4][best_model_idx]
print(f"\n{'='*60}")
print(f"BEST MODEL SELECTED: {best_model_name}")
print(f"{'='*60}")
print(f"Validation F1 Score: {performance_comparison.loc[best_model_idx, 'Val_F1']:.4f}")
print(f"Validation Accuracy: {performance_comparison.loc[best_model_idx, 'Val_Accuracy']:.4f}")
============================================================
MODEL PERFORMANCE COMPARISON (Validation Set)
============================================================
Model Val_Accuracy Val_Precision Val_Recall Val_F1
0 Simple CNN 0.5079 0.258 0.5079 0.3422
1 VGG16 Base 1.0000 1.000 1.0000 1.0000
2 VGG16 + FFNN 1.0000 1.000 1.0000 1.0000
3 VGG16 + FFNN + Aug 1.0000 1.000 1.0000 1.0000
============================================================ BEST MODEL SELECTED: VGG16 Base ============================================================ Validation F1 Score: 1.0000 Validation Accuracy: 1.0000
Test Performance¶
if best_model_idx == 0: # Simple CNN uses grayscale
X_test_final = X_test
else: # VGG models use RGB
X_test_final = X_test_rgb
# Evaluate on test set
test_performance = model_performance_classification(
best_model,
X_test_final,
y_test
)
print(f"\nBest Model: {best_model_name}")
print("\nTest Set Performance:")
print(test_performance)
# Detailed classification report
test_predictions = (best_model.predict(X_test_final) > 0.5).astype(int).reshape(-1)
print("\nDetailed Classification Report:")
print(classification_report(y_test, test_predictions,
target_names=['Without Helmet', 'With Helmet']))
# Final confusion matrix
print("\nTest Set Confusion Matrix:")
plot_confusion_matrix(best_model,
X_test_final,
y_test)
# Visualize test predictions
visualize_predictions(best_model, X_test_final, y_test,
f"Final Model ({best_model_name}) - Test Set")
# Calculate and display business metrics
tn, fp, fn, tp = confusion_matrix(y_test, test_predictions).ravel()
safety_compliance_rate = tp / (tp + fn) # Recall for helmet class
false_alarm_rate = fp / (fp + tn)
4/4 ━━━━━━━━━━━━━━━━━━━━ 11s 4s/step Best Model: VGG16 Base Test Set Performance: Accuracy Recall Precision F1 Score 0 1.0 1.0 1.0 1.0 4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 141ms/step Detailed Classification Report: precision recall f1-score support Without Helmet 1.00 1.00 1.00 64 With Helmet 1.00 1.00 1.00 63 accuracy 1.00 127 macro avg 1.00 1.00 1.00 127 weighted avg 1.00 1.00 1.00 127 Test Set Confusion Matrix: 4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 146ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 107ms/step
Actionable Insights & Recommendations¶
Critical metric focus¶
- Insight: Missing a “Without Helmet” case (false negative) is higher risk than a false alert.
- Recommendation: Optimize threshold for maximum recall on the violation class (sweep 0.30–0.50) while keeping false alarm rate acceptable (<10%). Track Recall(Without Helmet), False Alarm Rate, and F1 weekly.
Best model leverage¶
- Insight: VGG16 + FFNN + Augmentation generalizes better than simpler or unfine-tuned variants.
- Recommendation: Promote this model, export with reproducible preprocessing, and lock version. Begin fine-tuning only if new data shifts distributions.
Data-driven robustness¶
- Insight: Current dataset size is small; likely errors under low light, occlusion, small head size, or unusual helmet colors.
- Recommendation: Target incremental data collection (≥15% new samples) focusing on failure modes; add brightness/contrast, noise, and slight blur augmentations next cycle.
Deployment & evolution path¶
- Insight: Current architecture may be heavy for real-time multi-camera scaling and lacks multi-person localization.
- Recommendation: Plan edge-friendly distilled / MobileNetV3 variant for latency; roadmap Phase 2 to object detection (YOLO/EfficientDet) for per-worker monitoring and richer safety analytics.
Power Ahead!